The Anchors Hierarchy: Using the Triangle Inequality to Survive High Dimensional Data

This paper is about the use of metric data structures in high-dimensional or non-Euclidean space to permit cached sufficient statistics accelerations of learning algorithms. It has recently been shown that for less than about 10 dimensions, decorating kd-trees with additional "cached sufficient statistics" such as first and second moments and contingency tables can provide satisfying acceleration for a very wide range of statistical learning tasks such as kernel regression, locally weighted regression, k-means clustering, mixture modeling and Bayes Net learning. In this paper, we begin by defining the anchors hierarchy---a fast data structure and algorithm for localizing data based only on a triangle-inequality-obeying distance metric. We show how this, in its own right, gives a fast and effective clustering of data. But more importantly we show how it can produce a well-balanced structure similar to a Ball-Tree (Omohundro 1990) or a kind of metric tree (Uhlmann 1991)in a way that is neither "top-down'' nor "bottom-up'' but instead "middle-out". We then show how this structure, decorated with cached sufficient statistics, allows a wide variety of statistical learning algorithms to be accelerated even in thousands of dimensions. The Anchors Hierarchy: Using the triangle inequality to survive high dimensional data Andrew W. Moore Carnegie Mellon University Pittsburgh, PA 15213 February 18, 2000 Abstract This paper is about the use of metric d a t a structures in high-dimensional or nonEuclidean space to permit cached sufficient statistics accelerations of learning algorithms. I t has recently been shown t h a t for less than about 10 dimensions, decorating kdtrees with additional “cached sufficient statistics” such as first and second moments and contingency tables can provide satisfying acceleration for a very wide range of statistical learning tasks such as kernel regression, locally weighted regression, k-means clustering, mixture modeling and Bayes Net learning. In this paper, we begin by defining the anchors h i e r a r c h p a fast d a t a structure and algorithm for localizing data based only on a triangle-inequality-obeying distance metric. We show how this, in i ts own right, gives a fast and effective clustering of da ta . But more importantly we show how i t can produce a well-balanced structure similar to a Ball-Tree (Omohundro, 1991) or a kind of metric tree (Uhlmann, 1991; Ciaccia., Patella, & Zezula, 1997) in a way t h a t is neither “ topdown” nor “bottom-up” but instead “middle-out”. We then show how this structure, decorated with cached sufficient statistics, allows a wide variety of statistical learning algorithms to be accelerated even in thousands of dimensions.This paper is about the use of metric d a t a structures in high-dimensional or nonEuclidean space to permit cached sufficient statistics accelerations of learning algorithms. I t has recently been shown t h a t for less than about 10 dimensions, decorating kdtrees with additional “cached sufficient statistics” such as first and second moments and contingency tables can provide satisfying acceleration for a very wide range of statistical learning tasks such as kernel regression, locally weighted regression, k-means clustering, mixture modeling and Bayes Net learning. In this paper, we begin by defining the anchors h i e r a r c h p a fast d a t a structure and algorithm for localizing data based only on a triangle-inequality-obeying distance metric. We show how this, in i ts own right, gives a fast and effective clustering of da ta . But more importantly we show how i t can produce a well-balanced structure similar to a Ball-Tree (Omohundro, 1991) or a kind of metric tree (Uhlmann, 1991; Ciaccia., Patella, & Zezula, 1997) in a way t h a t is neither “ topdown” nor “bottom-up” but instead “middle-out”. We then show how this structure, decorated with cached sufficient statistics, allows a wide variety of statistical learning algorithms to be accelerated even in thousands of dimensions. 1 Cached Sufficient Statistics This paper is not about new ways of learning from data, but instead how to allow a wide variety of current learning methods, case-based tools, and statistics methods scale up to large datasets in a computationally tractable fashion. A cached sufficient statistics representation is a data structure that summarizes statistical information from a large dataset. For example, human users, or statistical programs, often need to query some quantity (such as a mean or variance) about some subset of the attributes (such as size, position and shape) over some subset of the records. When this happens, we want the cached sufficient statistic representation to intercept the request and, instead of answering it slowly by database accesses over huge numbers of records, answer it immediately.